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ABSTRACT. It is proposed 3 hsE (ho 3-D representation of ?n object is based primarily on a 
stick-ligure conjuration, where each slick reprBsenls one or more axes in Ihe object's 
Etnerahzad cylinder rep.^s^nUtion. The loosely hierarchical descnplion of a stick figure is 
interpreted k>y a special -purpose pr«£$tOr f sb!t Id mahtain two veelors ,-uid 1he &ravi1atiDnal 
verlical relative to a Cartesian s pact-f rami. ]t delivers intprmatiori about 1he appearance of 
these veclors, which helps Ihe syslem to rotalt i1s model into the correct 3-0 Orientation 
restive to the viewer during rBcOgnilian. 

This ra-pOrl describe raEaarch. done at the Artificial tnlelligenee Laboratory ot the 
Massachy 5 et1& tnslitute of Technology. Support far the Laboratory's artificial intelligence 
research is, provided Jn part by the Advanced Research Projecls Agency of the De-par tmenl ol 
Detente undar Of ike of Jdaval Research contract number NAGGi 4-75 -C -0643, ' 



Sam/nary 
3, \", i; Qbirrvcd Ihfll ihe ga-neralized cylinder representation oF a 3-D object generates two 
distinct problem^ describing the crosi> sec lions associated with each axis, and representing 
Me -'e'ative dispositions of the anas in space. 

Z. The second orobisrr amounts '0 representing tne spatial arrangemenl of a sfith figure. A 
mc-tnod for d&ii^ thii l* a iven - 

3- The stick-figura is described by a loosely hierarchical Assortional database, called a 3-D 
marfef. Use 01 this database is flexible, and it can s„ppor1 levels of description thai cover thv 
Spectryirt frpm a vary coarse overall summary to ve^y fine del ail af one small pari. 
4 r In prdar Jo be used,, a 3-D model has to be integrated Ihrougn an (essentially) analogue 
-nee nanism. caUed Ihe image-space proctor, jr. iH *n nin- a l iTptemenlation h this- processor 
maintains a representation 01 two directions (called $azii and Ji^mr} in a Cartesian space-' 
frame, in additional So the gravitational ver1i»l, 

5. Tha image-soace processor's instruction set is smalt, ]f$ important (unctions are: 

(a) setting, Ihe Saxis to one or the space- Frame's 3 axes or to Ihe gravitational vertical; 

fb) ^citing !hB SspasaT to an arbitrary orientation relfllive to the §*XH, this includes 

the ability to rotate the Sspasar about the S&k-.s-, 

it) Silting 1he Saxis to the Orientation of the Sspasar; and 

(d) rotating tha- space -U aire about four cisli^ijuithed axes, ts three coordinate-ases 

and the gravitatisnal vertical. Hh a m nirnal implementation OF the image-space 

orocessor, tne position oF the Sspasar would have to be- reconstructed after a frame 

rotation rather than being rotated with it) . 

6. The image-space processor can ciplivpr infomalion about (he lengths and orientations of 
the projections Of the Sax : E and Sipasar a ".to Ihe image plane, These help the system to 
"rotate" its mpyel into the cOrred 3-D oriental ion relative to the viewer. Son* evidence is 
given that this ear* be carried Out by a process of relaxation, 

7. Fahlman's symbol -mapping oroolerr. is dealt with by dividing *t into its component problems^ 
and using, special techniques tor each component. The problem et indexing for recognition is 
discussed. 

S. It is Observed that this theory may help to e-npla'n various aspects of the psychology of 
human vision. These include the "mental rotation" experiments of Shepard and his 
collaborators, and the dinrcal disabilities described by Warringlon £ Taylor (1973) that follow 
right parietal lesions. 



/rit "-.■nfJurran.n 

The two current ideas For representing Ihre*-diir,tn5icnal structures are- the 
"sclera ized cylinder" represenlalian proposed by T. 0, Binford and implemented by Agjji 
(1973 J, Nfcvalia (1974), and by Hollerbach (1975fc and She "muUiple. view" representaliOn 
(Minsky ]97S). The generalized cylinder ^presentation ol a structure it obtained by 
specifying, ils axis and the cross -seclian at each pom] along il. Agin and Wevatia used a laser 
raii3e-fin(Jmj technique So -obtain ihrt generalized cylinder representation ol such objecls as a 
barbie doll, a tnaka, and a horse. Mollerbaeh Eluded the .representation of a wide- range ol 
pQtlEfjf, The muJIiple view representation ie based On Ihe, insight Ihal il one chooses Ones 
priffiilives eorreclly <e.g. the "side" of a cube), Ihe number dF oualilati^ely different views of 
an object may be cuite -Email, A number 01 important questions of detail remain unanswered 
because thji i»a has not yet been implemented, and it remains So be setr wheShc-r a theory 
can be built uoon it. 

The generalised Cylinder represenSlon inlrodutes two main problems; Obtaining 
the axis and a descriolion of th* crpss-5*ct*0n 01 the diSFefenl parts of am object (arms, legs, 
torso), and representing t ne ^paFiaJ dispasHipri pf 1ho cOmpOnenSs 1hus oblained. The second 
01 1hese problems has hitherto received nD a'lenlian, and it is Ihe one thai we address here. 
To solve i1, one has to tickle directly the prOb'em Of representing, the positions ot items in 
fhrc-c -space,, and this article presents a method for dainj it which we OeliiVft- may be of 
interest 10 experimental psychologists. 

$ta.lrnnfht e/ jrV prvlit-am 

The principle ol modular design is central to the vision system of which I hie 
article describes a part. For ewampic, th c processes thHl define a place-1o*en in an im^ge are 
almost independent Of the processes thai subsequently group them (Mar' 1 975>; Ihe 
processes tr\a1 ?cec! iTE-nrs Sd be lested Tor sywn-clry are similarly independent t}\ |he 
routines that delccl it (Ivtar.- 197EJs the evlratlicm oi a form from Ihe primal sketch is often 
incepenoent of ihn processes thai deterge lhat fo'n {Marr 1975); and much of She 
segmentation of a form inlo its generaiizeo cylinder descriplion tan apparently proceed (to a 
first appro* imal ion} inoepencenJ y cl knou'lB-dge aboul whai that 1orm is (taarr & Valan. to 
appear). The repre-senlalion Of the 1hree-dirrSrtn*onal s-trucSure Of an object using jene raided 
.cy|ir*ders can also be spU into (ho- 1wo problems mentioned above/;, and Our first proposttion is 
that the two problems are dealt with by separate modules. One module computes the 
description Ol the shape ot each component, and another describes 1he rotative spalial 
dispositions of HlOse components. 

From Shis prooosilion, M fellows thai describing the spatial ditpOsillen of parls 
Of an Object Gr amnial may oe recutec ;o The prcblem oF descrioing Ihe dispositions of 1he 
axes that occur in. its generalized cylinder detCripliOn. Thus for animals, our problem reduces 
to thai of describing stick fjrjunrs - models made ou' oF pipc-tica-^rrs, cme For each axis fsea 
figure 1). The virion system being constructed a1 Our laboratory is already capable ol 
computing this description fro-m a raw image in simple ea^cs, 

The problem then it to represeril the three-dimensional configuration of a 



FIGURE L The IhaOry *s ier 1 S NijH thu 3-D rtpTtsanblion of a rbace it decomposed into Iwft 
parls, the description of ihe tras$- Bac tr6nj lhat Kcur in the shapes jtnerflFizad cylinder 
representation, and the cksposilio-- el ihe a«es of these cylinders in spue*. The theory deals 
with She second problem, whfch is essenlially Ihe pr&blem pi dercrib.ns slick figures, TJ» 
*hapes in these pklures **r B mads out el pipe-cLeiner*. the r^.r w i|J ha** m IrOuble in 
recognizing Ihe giraffe, daer, rabfeiT and ulrich. That Ihelr rscoanilion "is s& *«y «* K « it 
reasonable tn Appose th»t at same *{*&, we ouriel„ ei decorate 1he 3-0 r*pr BBa nliti Drt 
probieirt rnto similar COmoDrenJs. 
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F!SU3E 2, This is the raw data provided :o our system Irom The intermedials visual processpr. 
IE consists -of 4 collodion of imagels which ar# descrppiions of individual gen* raized cylinders 
round in [he image. E*ch im^gei has 1wd end poinds in |ht imaja plane and" cptipnally a shape 
prnpe^y such at (stick which supplies additional inFormalipn about th# imajei such as 
average thickness, royndedness, fatness, and *o on. 
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FIGURE 3- Thia diagram summaiiies our overall view o1 !he Tecogn'l^on proolcm, The 
nnpOrlant points For the present article are <a) lhat 1he rapresenlatiOn 01 3-D mo&ets rs quite 
separate Fram She represent alion fl' (yntl itna semantics (b) indeves ey'isl thai give rich 
access to lu^dional semantics panniers Iram aescriplions a' every &tage alter the scpsr^iiDn 
o s figure irom ground, a/id (c-) 1h*t Ipr difficult images, considerable interaction may have to 
taite place bs-'tfeen the description nf t Form, the image '-space peaces son and the 3-D model 
ndexer oe-ore ar appropriate j°-D rrcdel ie fa-nd. When o»w it found, &till another step CoF 
reliiftlian] m?y be nita**pry before 1h* fundiOna.1 =*rni.n1iei indexer acquires enough 
information tn recover the correct pointer. 



stored slick figure so as to relate lh* angles and apparent lengths four*! in an image like 
figure £ Id the three-dimensional structure m I the Object and th* respective from which it 1* 
being viewed. We warn a solution 1hat if some sense- minimizes the coirtp-utatiOnaS complexly, 
but net necessarily the CQmputaliOnal power, of the machinery required to implement if. 

fl/iciiftaulid: rer.ofr?\ii!tfa it not in ait -<jr-tMMtfi pT-OTfTJiJ 

Bo-fore siv.ng an Oulline of the ItiBOry,. we need to make two general points 
that have decplje influenced 1he way we approved She Jecognilion problem. The first is I hat 
the stored throe-dimensional nrpra-sentalinn of an Object is separate from the representation 
cf i1a functional semantics. This article deals only with the three-dimensional problems t hat * 
horse 1rots, ga lop;., mals grass, tan be ridden, and is liable to kith are not representee hero. 
11 is very reasonable to keep the Iwo separate, because a Irving horse differs in • 
fundamenSal way from a statue 0< * herse, despite Its sirralir geometry. Nevertheless, there 
are grounds lor thin*ing 1ha1 the top-feinl token thai organ*** ih H functional semantics, of * 
horse Lj the One that is- closest to Ihe linguist label "horse", end part of what wb mean by 
recognition is the ability to address- this pointer 1 an viewing a^ image. 

Which brings us to Our second point, [t ; S often the case lhat the functional 
semantics pointer can be acquxec qu.fo early in in* analysis ol an image - many simple and 
delmile cuss exist that ran be evaded beFOre a 3-D description has been built. Any 
indexing strategy (or iasl recovery of the iunctional semantics pointer would certainly take 
adv*nlage of this, whith means being sensitive fo descriptions at every stage alter figure- 
ground separation. The varjply of cues that ire available in most images probably means thai 
only rarely will one have to proceed all Ihe way to a 3-D description befo/e a match in Ihe 
diabase is found, Indeed, we would ckje-cI this Id happen Only when the object i; being 
seen from a deceiving persoectc/e, or w.^n Ihe prevailing illumination is- unusual, It is quite 
easy to design flexible indeicrng techniques, that can mate use of clues OF diverse kinds frpm 
different levels. 

Our Overall picture of the recognition problem is iltusl rated in figure 3. This 
makes clear our belief {hat there are many ppfhs to Ihe functional semantics pointer, some of 
Ihem fast but not necessarily available from every image, and olhers (hat are slower, but 
Which usually guarantee results. In a penalTa1ing analysis, Warrington ft TayJor U&73) 
concluded thif the 3-dimensional description and the functional semantics of art item ere 
represented in dislintl cortical areas. Their evidence for this assertion is double dissociation 
between the two kinds of deticit, observed in patients with lift (for semantics disorders) arnd 
right (for disorders of three-dimensional represent *1i0n) parietal lesion's. 

This article is ronce'ned wilh only one small part of figure 3, namely Iho 
construction Ol a 3-D mooel Of the disposition OF the axes in space, tn order to be convinced 
that this h certainly one of the possible paths to 1hu functional semantics pointer, on* has 
only to look at figure 1. Pipe -cleaner animals exhibit only the lengths and disposilions oi their 
axes, yet we have no trouble recognizing the gir^'Fa* rabbit or ostrich in 1h:s figure. 



RcitJJnff »/ the thwry 

There are four main cc-moon^ls Lo I he IheOry. We give a brief description Of 
Ihem first, id thai 1he re?dar has an overall 1r»mewor* wirhm which to fit the cfelails. 

Al some sta ae jn 1he represenlation of Ihree-dimensionai 5pace| ono ^^ ( 
prrimt.** Ability lo represent a veclor fi.t. a direction and a lenglh). Accordingly, Ih, first 
component of the theory is a processor thai provides thr E primitive ability. ]t .* called the 
iffl.tajrc-.pner jwutftr, and it can m?in1ain 1wo connected vectors within a support, ng space - 
frame. Th*Se two vectors are «Ued the S«h and Ihe ft, J( .j„r (an abbreviation for spac- 
er row*. Tb* processor can translate the end of Ihe Sspasar Id in ass-gned pos-lion on the 
iaxis, can, rolate il around the S»is. and can re1»ti it in the plane containing both vectors. In 
this *ay, the Sipjsai can he brought to »n fl rbi(Tary relation with Ihe Saxis. 

In addition 1e these families, the image-space processor tan move the Saxis lo 
wherever the current Sspasar happens 10 be, and il can **\ , 5 though a small number of 
soace-frame rp1ati0ns coLJd be parfDrmsd. We do nol regard Eh, B pace-trams rOl aliens as 
true extra facilities, because Ihw are w^ys ot simulating them WS mg only |he Saxis and 
Sspasar. 

'id Ltefglness of the processor for recognition arises from the fact that 9* 
well 4S maintaining the three-dimensional relalian bilween Ihe Saxis end the Sspasar, it can 
compete She lengths, directions, and angle be I ween their project ions. The computational load 
attached lo doing this is small. 

•The second component of |he theory Is 9 prepositional database fhal 
represents by a* tertians vselul three-dimensional relations between Ihe axes of ihe Objects 
being viewed. The dalasiructure for a single physical objed is called 4 3-D iW r J, jind its 
purpose is to explain every image Yemeni delivered by earlier visual processes, up to « level 
Of detail appropriate to Ihe cirtumslances. There are two Importanl poinls about this 
database. Firstty, its organization is loosely hierarchical. It can prou.de description of parts 
Of ^ efcject that <over richly the spectrum FrO/n a coarse, Pne-anis descriplion 01 a whole 
object, to a fins specification of n** small pari of it. For example, al the top level, a horse 
may Oe representee as a single, ho/ijonlal axis. At a lower Jevet, Ihe two dorelegs are 
Irealed as » single a*!*, Al Ihe nc*l slage, Ihis description decomposes IP the left-tOreleg 
and th* right-foreleg; and further do^n, Ihe single axis description of the lBT1*rore>CE 
decomposes to two, splicing at wh»t the laym,n «odd call [he knee. Figure A shows s&me or 
the *ays in which a typical animal ditaslructure COulrf be decomposed. Th* second point is 
that, three-dimenj.or.al positions are reprtunled by local relations belween adjacenl parti of 
a body, not oy absolute coorOinales in 1 circumscribing frame Pf reference. Thus the position 
Of a toe is scored relative to a too!, *h.ch .s stored relate to a leg, which h iLrrn rs stored 
relative to the torso. In order to discover the ulalionsbip between Ihe head and th* tOe r 
these intermediate relalions have (0 be examined. 

The Ihird component of Ihe system is the tntorpmcr, whose job is to create 
arid maintain the interface. beWn Ihe database, the image-space processor, and The 
Information bein 6 delivered from the image. The inlerpreler is capable- of reading Ihe 
asserlionj in a 3-D niOdeJ, binding the Saxis and Espasar in the image-space processor to 
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FivUHE A The rep-resenlation Of a ^1icX fi&ura I: loosely hierarchical, and allows descriptions 
to he created at many levels n1 detail. A.1 1hc top level, * horse may be rgprcser-.ted as a 
srnjlfif hD.'izonlal axis ',\o answer queslion? liu "Where is 1he horse pointing"). To answer 
Thi qt«s!i&n "Where it Its Front lefl hoof pomting?", the lelt ioreie£ will have been unpacked 
to a considerable oeg-ee 01 detail, while IhS hir.dlejs may slill be bound to nolhinj finer tnan 
a sin e;1* HlNDLEGS axis, This figure shows some of 1ha ways in which a lypical ANIMAL 
flat as.tr yet ure ma/ he- oeeompased. 













I I!.;!.-: -. \r-. ! *:.-!', l-ikr- IT:t ^iriky ,v rj: L .; .j972J, illuslralCS 'he i-.? i,nrp Cn 4 ar ;i*ii 
■an 1he aescriplion af a f euth. [n or* ro*, "ha snipes ?re 5e*n as squafei, and In (he other, 
as diamonds. The Bslablishir^ &f are* irt 4 ?-3irtertiional ligurt is imporLanl lor Our theory, 
since it determines, ho-*- l": description of a £-D r.on^ifljrat:Dn is constructed. This h'gur* is 
In* 2-0 analog &f figurS 1, sir-M il ist*fcliih*s th*t one p-recondfiSicm Fo-r using our theory as 31 
psycKolOE'c*! made) - n»mdy the iomputalian- af rues during the analysis *f 2-0 p-ntterrts - is 
satisfied by our visual systems. 
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FIGURE 6. The printtpiB diretlions deFined in the tepit are displayed graphically m fhe figure. 
The- imjgt-EGijcB proceLfiDf i? sapabl* »f simulating roSalions ahoul Sup, Slront, Shor iiorit*l 
and S^e-ftital, Aflsr eucK a relation, 1he po$ihon fl' the Sspesar Cand possibly aiio of the 
torn) must be re-tens trucled from adjunci relttions \& tf* 3-D mede'. 



appropriate anes in |he 3-D model, and causing these vectors to be rotated until 1rwy have 
the Mnft 3-D relation to o*>e another as if specified in jh* mode!. Tne interpreter can 
COmpro the risuHing vector with an image element, and can report on any disc repine ins 
se£w«n predicted and measured properties of the ™*ga. Various global variables are set 
and read by the interpreter! Ih»y include a certain degree OF trans lational freedom, ,r*d an 
Overall scale Fidor, calied the *Wr, which governs She reUlien between the size property 
Of ah item in 1he database and 1h» lenglh of the Sspasar 1o which il E ives nse. Daemon* 
froOul which 3-D model to instantiate and whal parls of ,| ro concentrate p n torm |he 
control Irng information tor The interpreter, which |ho-. t-ies to mrtch 'ihe model to the imago 
by three-dimensional rotations. 

The interpreters main requirement is thai it be able to mOve^ around the 
data structures it initiates in a flutnl and agile manr^r. This is necessary becausa Ihe 
com pg? agonal resources in 1ha image-spacs- processor are limited to representing at most Iwo 
vectors at Once, whereas 1he range of questions on* needs to e« able to ask of Ihe whole 
system is large, for example, in prder 10 answer the quasi ion In which direction is The horse 
pointing?-, Ihe image-space processor has to be hound 1o Ihe horse 3~r> model in a completely 
different manner From lh»t required to a^war "Where it it* FrOnl left hoot pointing?" M 4 
parlicular instant during a slep. Several interesting issues were brought into focus by having 
to design a satisfactory ifflplemsnlaliOn of the interpreter. 

The Fina( componenl Of the theory is something we call Ihe rclavenioti 
hyfiothtaU, This hypothesis states that by using the vinous cues available from the image, 
including information about o&scuralibn, lighti^ *nd support a; well a5 the lengths and angles 
Observed (here, if it usually possible to align |he image -space representation oi a viewed 
Object accurately with Ihe object 1 * real-world orientation. Furthermore, this may be 
accomplished by a relaxation technique; thai is, al any inslanl the compensating rotation is 
made about that axis (of the four available in Ihe image-space processor) which redutirs the 
largest discrepancy currently mc^ured. We cpnjedure 1haS thr 5 skalegy will converge. This 
component is siill a hyuo^hesis because we have not yet finished implementing it. Our 
expectation is t however, that its irr.plerrentatior, wilT strongly resemble that of Ihe earlier 
visuaJ procusses with which we hf, t had experience - i.e. it will consist of a cons.derable 
number qf specialized diagnostics thai interact in a Jiirly -simpte way. 

]t will be evident thai the image-space processor is inherently powerful cr*ough 
lo repose*! two-dimensional palte-ns, sue;- a; r.- e configuration of features on a face. Such 
pallerns may be thought of as degenerate esses in which the girdle-angle is zero. The only 
requirement is lhal these patterns be described in in appropriate way in the database. This 
means that axes nave to be set up in the two-dimensional pattern, and the configurator 
have to be deserved in Ihe usual way relative to those axes. Intereslingiy, i| h« long heen 
fcnown that trie cho.ee of an »xie in an im^r, can greatly influence the way in which shapes 
are described. Figure 5 (MinsKy & Papert 1972) sho** an sample of (hi*. Important 
modium-level vision noculas ke symmelry binding (Marr 1976} can be IhOught of as. helping 
to find the axes !ha1 it \r, appropriate :t> „se. 



Sm(i;if>-xyjnr.t , [ii'(M*'ff-iH' 

We begin the detailed acWunl of Lhe computational !i£;lilies attached to each 
pari of ttli theory by diseasing the image-space processor. The interest here ties in 
minimising the cOn-Cutalional pO*e* Shat On* g*«- We require that the processor maint*in* 
for simulalcs tne maintenance &F> w directions. They are: 

(01) S5PASAR, which in lh* minimal implementation is the only vector lhat can be rotted. 
(D2> SAX: 3, which i* lhe vector to which the Sspasar is ittithetf and around which II rotates. 
These two vectors are maintained in 1 iocal ipwe-ffame, which may be thought 0< " hemp 
delined by Ihrse directions: 

(03) SUP h whrth initially coincides with She gravitational vertical^ 
(Dfl) SFROr+L *.j. Far a horse, the dlrecSbn in which it is polling, and 
(□&) 5H0REZQNTAL, which is perpendicular to Sup *r.d Slro-.t. 
Finally, because i1 is :in impprlanS direction, we need 
(06) SVERTECAL, which is defined hy the jrayitaliOn*] vertical. 

The instruction Jet to the proceasor divides into four parts, 

(a) Tne S ax ^ initially coincides wilh either Sup Or Sfronl, {e.g. S u p for a man, SfrOnT for 
a horse} and can he reset to 'heae direction holders, at any lime. 

<b) The fcpttlf t*n be attached tP the Sajci* at a opacified po ; nt und ratal cd around it. 
The most irr.p&rtanl three-dimensional rciifonship between the Saxis ar^d 5spasar ,i 
called an flrfj^ci jtfeifon, and it it writlen 1> i gX p is the pdrition on the Ssma at 
which the Sspasar it atlachedi J is the Wiium*™ d the Sspasar to She Sams, 
measured in the plane lh»t contains 1he-n both! *nd. * lhe ^rdfc-OHjrle. Ascribes the 
rotation of lhe $sp«ar arc-und tne 5a*i* {see figure 7). Ths- Ss.pas.ar can also be 
translated a*»y from the Saxi* according to cer1 8 in roles. Thit makes it possible lo 
represent the lect tnat one's arms ar* np^ altached directly 5(5 She axis o( cne^ tor«, 
but air* translated aw»y frpm thai jwt. This Sranslatinn is carried out by means of an 
frofrddiiij- relation fi ^ where «t is the disFanee and « the girdLeangtc shown in lisure 
3. in ^he uatastructures exhibit Bd late* in lhe a*ticl*. t adjunct and embedding relations 
are combined; in one expression,, which, has the form (p fi jrJ W ^ 
■ (c) The Sari* can be -eb&u-d Fa whatever the Sspasar I* currently Bound to, and in so 
doing it assumes lhe spatial corjfdinalts p1 the Sspasar. 

(F2't Spacf/iwrrio eprr etsoiu 

Th* space-frame ma* be -otaled aociut any of the Four di*etliona 4up, SFrOnt, 
ShQrizOnSal *ai BwTial fte* figure ft ThMB operalions are called respective^ TWSFL, SPIN, 
TILT and VRDTME. !n a nHntmil implemBnlatlfln, which is interesling for reasons we shatl 
discuss Uter, tKeculinfc theae rolaliOnt would use the sarna machinery 1hat rotates the 
Sspaiar ahoot the S«*is. He.rrC€ c«eu1ing a spate-lrane operation would cause the current 
&s P asa.r lo be lost, having to be r«OnVr«tfed alter the ft arnelrSnSlormatiOn. If the Sexfe M 
not aligned with a n »*i* of ^ne Bwe-lrame, it too will ha*. 10 oe reconstructed. Became 01 
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F[GURE 7. Th# mosl impcr'ar.f 3-D rsiatiOn&hip ie called an adjunel ratal ion,, ip, i, ft). Th» 
pflSition p, the inclinsiior, i j.-d the girdlfl-ariglB- g 3»s te'irltd in Ihe text, and illustrated herfl. 




FIGURE 9- The oriental ion oS the irilsfnil eonrdjnale system depends on 1he an^le of gaze. In 
order to minimize Ins computation*! power required tQ conpule projections in the irtiage 
plane, s vector^ ^ompanenl towards the viewer '* represenled separately from its tomponent 
in the- tangent Diane. As 1he 3^=1* of gaze changes, 1h< eoordlnale system it related restive 
to 1he Outside wOddL If the viewer V fdbh& sleight ahead, the inters! coordinate system 
used is shovtfi*. if, the figure as (a, t. «>. ]f the viewer moves hi-S gaze to ascension a end 
CDt'iifllii- r!. 'n- (S.::r..:r^sr :.y;:£^ ij rO-:li-c -a .iccJ :hc w Axii., anc 1, sbo-i* 1he new 
direction Of the n Mis, resulting in the frame <^ y- => illustrated here. In this ™ay, no extra 
^or-j.;:. J :iCrl il ■■sdiie-l j' dr :•■">:■ la ri(;1i;r,T.'-.£ '.'£ |v:i,f! :.:.;:■', 3" i CClcr in I - rrajc n.iir:;. 
In order tor Ihis 10 be possible, 1he image^spste processor inquires accurate ini&rmalion 
ab&ut the direction of the ang'^e d! gaze relalive 1d the gravity ional vertical. 



Jhis> it is nearly always advisahle to =s1 up a new spm-fre-me when the 5s.<is is abound to a 
new part nf (he 3-D model, [1 fc worlh emphasizing rKal m 1his Iheory, iniormalion aaoul the 
currenl dirsdion of the gravitittonal vertical restive to 1he vaewer plays an Important roie In 
defining the mlurnal representation af 1h B spalial dlsppsih&n at a 3-0 model. 

{^3,J Comjimbijr pnjttiiom 

In ewdir to compute the appearance or a vector, one has to Know the viewing 
angle. The nalurai geometry associated with an imaghng system is spherical When Ihe .ma B e- 
space OTOce^or coTp.tei (ha lenglha and angles associated wMh the S,*i 5 and Jspuvr, it 
therefore has Jo t«k« account of the direction of e *ze, This carr be accomplished in, | w0 wavif 
either an Jnilializing. pair af rotations k made, about the lup and a&Ou! the Slront, equal and 
oppose *0 :he asa-^P a,-.; cecHnal+On Qt the viewing anglei Of (his transformation is 
carried out On each yettOr just hefpr* Hi projection It read oft. 

The first melhod Es .simpler. 11 hat the virtue !hat if the space-Frame Is actually 
represented in an internal roordinale syslum 1hat spe;ih*5 rb* comooncnl in (he direction 
towards the viewer independently Of the component in Che tingitnt plana (see figure Si, I ho 
'^qj 'ec o'ci:.-i.l .:n; a-e av«,lsble *Hv:ut t-xf-a rvnoj-oiion. LiLr p-ese-,1 ryr-.rnrv s 
irepltmcnlea this way, 

CjP+J ftafUfelfeji 

Alter several chanse* of Saxia and Sspasar, One can find thai the vectors are 
being constat tad a considerable diiiant* sway from 1he prjgrn Of Ihe ima^e -Space. This is 
not a problem for a computer imnlemenSaliOn OF the theory, bu1 il would have to ba 
considered carefully if the (htory were eonslfgtd as a psychological model. 

Tht! dfitttba&e and ft* ijTtffrjursi*r 

The Overall picture of the dataslrudure that ■$ ™1 up J or » particular imae* it 
oF twenty or IhJrty indapendem Atoms, each special izng in the description of one aspect at 
the Image, for example, atoms night be sel up for the head, twso, tail, Foreleg, lefWareleg, 
hoc*, iHfl -frpnt-hOOf K horseshoe, neck, mane, and pair -dt -ears. (Jlher atoms describe lt>« 
shape Of 1hese Items; for ewmple, Ihe tail may acquire the description -slick". Ihe lorso, a 
"cylinder" Special atoms will describe the lektuce of the tail, Ihe colours of She v-arious parts 
(the specific colours that Occurred here)., and the sheen p n lha animal 1 * COaL 

Each OF these alOms r-anies a particular :ype o! dalastryctuns, and contains on 
its property lis) for the default extension of its property -list) values and relations, appropriate 
to that type. For esrample, a ;orso-aKis-a1om h»s adjunct relations wilh Ihe a*e* that connect 
to it, and a pointer to an a1om fo' tho larsn'a shapft. The shape A.tCm, "cylinder-shape", 
Specifies the Je^n an ^ ^i^th OF tJ* cylinder, and points 10 any modifiBrs it may ha^e - like 
bumps on it, whether it is tlaltaned, ancf 4 description of the direction >nd doEree to which it 
is conical. 

Afte; strulinizlng «n image, il will be evident thai the resultina dalastrmiture 
C*n become very large. Finding the required informal ipn in it - i.e. evaluating a rele renew 



within it - it there Fo*e a major problem, We approached Fh# problem by designing a system 
in which freedom 01 rdercntc I; a tua-it system facility, and by decentralize re; ^r- lar as 
possible the mdeaes thai support I his 1reedoTV r or giot.dIc, 1he lop-level HQRSE atom may 
be accessed directly i1 tha system is asked (0 evaluaje 1h>e reference Shorse; but the 
poinlers To The parls 01 that hQrse are kept in * subsidiary indeK at Ihe particular horse atom, 
Thus nl order to discover the alQn thai stands tor this hprse n s lail, the alom for the horse 
must be interrogated. Ihe horse-atom therelgre acts line a Intel index - a management 
function that is super imposed upon its dal a -storage ItinctiOn - and vi call such » local inoen a 
pn^Jirtt. Seing a packet is mandalory For physical objects (PHY509*), but optional 1&r lesser 
strudures. Any aloT can however become a packet, and it doe* so precisely whenever 1he 
viewer's interest in Ihe image is- suslai-cd lenj e'OUgh lor the pieces uF description that lit 
■"below*" that atom to become inslanrjalcd, It it also possible 1d -create a packet lhat 0Tg,ariiEe.5 
in a new way parts of 4 description lhat have a' ready Oeen instantiated. 

The tymhal-- mapping ftrehfoni* 
In order to eOnsirud mechanisms 1hat would allow these things to happen, we 
hid h> design solutions to a complex o1 problems that have cortie la be known as tymhnl- 
mapping (Fahlman 1975, UcFJermOlt 1&75J. When he (irst introduced the lerm, Fahlman meant 
the phen^mcnsn wherry bei^a 1oid 1hit Ciyde is an elephant causes much general 
knowisdER j'aou? elapnanls Id apply SDecifically to Clyde, Because much of the r»st of 
Fahlman's discussion concerned recOgnilion, there has been some confusion, about the 
relevarce o' this Dhenorsnu", 10 recOgrntipn, There is no a priori reason why Iho two should 
be connected at all, There are in Fad 10ur qij te tfisFinct problems involved (Warr 197:1b). and 
Ihey ace all important for systems lhat us* stored descriptions to account f&r new data. The 
lour problems arc: 

(a) The one Fanlman nrig.in.aUy addressed, namely the application of general knowledge About 
elephants 10 Ihe specihe irislance Clyde, which occurs when it becomes krjiwn that Clyde is 
an elephant We Shall refer to 1his as Ihe pwperiy-inheHtanEi) prohbeWj because here the 
tas'* Is Fo map properties held in 1he daFabis* onto * speciiie instance oF a known class. 
<b) Suppose tha^ Clyde 'Ihe elepnant has already bean menHoncd in Ihe current environment. 
Given only the reference AMMALOr URC£ GRAY O0-ECT r or PEAMJT-EftTE^ use this to find 
Clyde in tha daTabas*, W* shall tail this the mfcrcUtt-mUidoin problem, since in some sense 
Iht issue is how widfr to mine Ihe window through which you can. access a current item by 
"escribing its properties. 

(c) The raference-w^dOw problem merges into the third problem, thai at inrfexinjy /er 
rcirojrjififlia, bu1 there is a distinction that is probably worth preserving. ]n tF*e reference- 
window problem, the ilems \',i\ are 1o be referenced arc already present as instances in tha 
current environment. In recognitor., Ihe problem is to access i su'.tabie templale from Ihe 
dalaoase with which to describe some incomir.j in'or^a^on. Recognition retain in an instance- 
Of some tempi ale j whose Own fefcrrj.nc.f-windcw problem 1hen begins. 

(d) The reFe^ence-windciw proolem meets the recojnilion problem somewhere near where 
they both turn into the fourth problem, the prvlAum af rm-.alL The diFierence belween the- 
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FIGURE 30. Our mtcm*? re„r flS e,rat,pn of 3*D ih ws is maintained on She pr* P erty-F«t* c f 
Iwd kmd 5 D f atoms c*ll B rf templatBE and S^w*. Tfmplades ( ypBfl r cafi « nanies ri k* yOl^EY) 
& tor* mtornaliw abogf archelyps shapes *h.l e 5aiDn 5 (n a . T eL ai |h e Id-ih $|??6> m use-lI Id 
represent parties m t |,nc« of the- temples. A third kin<J pf name , S the SreFeranca 
(lt ? pJate nailB! prBti , ed ^ 3 S) whFch Hppfrar 3s vi|uss flf prDpBrlJK jfl fhB |E Wm 

aho**. These are decoivfecf p 0inlers thai indiule Fhal on * sh^kf Mrs! „ e if | he particular 

rCC ' e ' ,Lt h3i b * tn ^l^^tBd in th» current ^irbmn^l beFar* felJowing through Id th» 
mdiCiied Fenplale. 6 
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FIGURE 11. Thi& figure )Hurlr*t** th* proaftGM invokvid in asking for I ho Ihichrrass of the 
torso, SI 876., Informal ion fhal will be u«d tfl »n$wer t^is- request is shown in (D which is 
p*r* of The environment a: the li-ne d? 1he reqt*s1, (2) shQ*s the- $1cps lihcn tf> answer the 
request, »nd (3) shows Ihe elFec! this process has on She environment. 



recall problem and the rajJerence-windQw problem is Iha1 m the recall probtam,. Trie item (0 be 
accessed tias rs&t yet bean braugM into ihe current envirQnirenl. The difference between .: 
and \hm recognilion problem- is that 6ns is not simply recognizing yonder ponderous gray 
object at ait elephant if rs * specific cicphant, namely Qyde, Iha one *no a1a your fc,^ *F 
psar-uls. last Thursday. 

Sottin dr.finiii&nt 
Only prdbtem £a> fta) and fc} will concern us in this article, but before we can 
exptain Our solution* to I hem, *e r^ B d to inl reduce the folding delinilions: 

(1} Template. 

A lenp-ale, denoted by an upper case name like HORSE, Fs an archetype 
property-list. Values in this p;ap«rty-ikt «ie expressed *S SreFarences {detinitian O) below*, 
nat as p-aMicular instances, 

(2) $ntom 

A SatOm, a.s., *L77&, represents an inslance of a template, and is created by a 
process carled imvntitiit*. A SalOm hK a property-list, m$ Ihe Ian-plate from which a Satom 
i* derived is stalled by the SCLASE property 0' She Satom. Properly names pr fl fi*ed by a S. 
li*e ECASS, cause tneir values to be quoted rslhg.- lhan evaluated [[ n roughJy true th»t 
valuts not sp.cifierj on >h* Satom'* property-list dd?ult IP vatues en Ihe SaEom's template 
{Raphael 1SSS p. S5>, b-jt to hat actually happens diife?s in an import tn! way from a simple 
default. We explain] his below. 

(3 J Stvfmnaitit 

A Srefereftcs j t a name, prefixed whh * 5, which may be evaluatnd (by EEVAi.) 
in the current environment, A ^eFerencc evaluation is successful if it returns a Sa-lOm. 
Typical Wlarences are 5ANEMAL, SH^RSE, iSHAPE t 5C0L.0R. References OF the form 
fSre-fe.er.ee 1 . 5'ef,s- B nce2) art also allowed. En such cases, the result ot evaluating 
Sr e lerente2 helps to ctlflftt Ihe context In which Sreforencel should be evaluated (e.g., 
(Storso L Shnrse> Or CEbump . (Snack . Shorsejft 

(fl) Pattlm 

A packet Is a Salam that cdnlainj a |-o C a| inoex, usually for some of th* 
substructure* of Ihe flb.ect thai the SaSc-r represents. Fp.- example, if Si 776 has SCLA$& 
HORSE, one 01 the entries h ;:s i-.drr* might be (7QR.5Q Sifi76r, meaning thai the torso of this 
particular horso was ^prese^ed at the top. 3 BV el by }h* Satom 5187$. Any Satom can 
become a packet if 1he items thai it organize* are inslanlialed, 

(5) SliiJe* 

The Hinde* is a general refe.-ence irrdftx.. One can address it with an evakjable 
reference, snC :t -,n entry evi*;« il *i.| ,-ehjrr, aether rderence whicbwill help in evajualing 



Ihe nnaintt. The dnlincliDri h*t w «n th„ jinde* and [he local pa tK e1 inde^s p* lh,| lha 
S.nds* , s a permanent body Si gen„ral knowledge {Fike a dicUunery), whereas each Idc-1 
packet i* .tay. specific, and uuully of orfy , emprj , lrv i BpoPl , n «. Be „ u „ ||w KfKte 
cons.st, of , se , 0( mm facts, it is flllitfl Bliy tfi add |o it . ^ ^^ ^ ^^ ^ 

permanent «■, be flowed ( £rDW VBry el ^ rste Af preiefitp wb ^ ^^ ^.^ 
under the Opiates they concern. A Mom represent* a specific Stance mjiy he + h0usM 
a( « bem B pilled in to- the General knowledge twined in [he fl iHd „. The plugs are 1ha 
en tr.„ ,n (he Jatarf* WS «izing w-pstkete, became il is throat, toes, Ihat ■ twhlrwce 
returned by ihe Sirwle* can evaluale to to*! Satom. 

When a S^rWnce ,s made, it i$ evaluated by Ihe junction SEVAL which is 
deigned >n accord,.-^ w f- toe principle oF least commitment. [< a foterefl « j s nade . wirhir a 
COnlnl th.tn»l». IH. rif m < unique, (h,| refect is return^. For „ BMp |« r i| «, ltoffl 6n 
Ihe StOrsO ato*» OF a horse refers to StaiL, 1he ™i*ren CB eva.^1*s to i^, jW^, tai ] This 
evalua ion would be unacted by the P ,* SencB of „ tt HHWTOe in 1he ]rTiage . „ n0wfiljflr 
Ihe re^rence was made exlerr.a.ly, (tor example by pari of th* dW«1r u et urH - tor a &0up 
reap*}, the Stall reference would evaluate to an ambitus par. Other knowledge wouid have • 
EO bo deployed to r« J V e to» Jffl fai su | 1v , se VAL ^^ ei(terisivB Ufie ^ EN? Mvm9Vim jh the 
Ctirfin! environment and in {lift Sinde*, 

ExamplK of the dalaslfUEtures deFined here appear in figure UX 

S»m* , ( ll ! te ? l9d ab&V * lhBt V * lkraS thlt * re nDl specified ^ ^ pPflP-rljr-lis.t ot a 
5a. om dBfault 10 the v„: u « oh 1r e pr p«r|v-1iM of lha £.|0mS tern P l a t B Whit actually 
happens dif?er S in „ imports Wly j rfirs 1hit . The v ^ fi [hat ^ Cur , n tN f (f &( ^ 

empLato are (or contain) toUrta**, nol tpttifit Satoms. ^r.Fere^e, canr^, be u E Bd Plhar 
thw * r^ranc^ t^v mutt be «p|»«d by S p B d« C i«l«s -ie by Satom, p beforr, the 
a 3!0Cf ated t,^p| a f a -, ^op^t y . S f s .[ Mn be BMni . nBC . ^ n „ n a dgfauM t(? a Srefer , cncB 

tai . the ^ II the reference ha* no relertnl in toe eurrenl environ^!. > suitrtl . fi n e 
it instarihaled. 

^ fh , » ^ fr " mPk Wi " he ' P t0 d3ri(y ChE ' SL * WW Ch,t "S^-hM SCLASS TORSO 
and tha [he SHAPE told under Ihe te mp l,[ a TQR£0 iB , he reFflf4fK( ^ORSC-SHAPE When 
w. evauito the SHAPE ^op^ty ., 5 i a76 , lhe eva;u , !or , bIIs thr ^ h i™*™^ 
encounters th, ¥ Sr e rer C ,, e . H 9 SafDnl , 0r lhia „ rBtf5 jh , pD j|re ^ ^^ , f ^^ ^ 

^en md^ed und.r 31S75, & a n* w 5atam ££1576 ^) * insl.nhaled horn lhe TORSO-SHAPE 
templale, ind SI976 is *ntof B d as (he SHA?£ property of 53 376, S1976 stands fcr lhe 
p^l.c^ar shape of tois p.rticul« tor Efi , aHhou E h as i| i, „ QW , S L g 76 COrisists mott| of 
•taut* through to ,t« t^plate. The e ru™i Fdea^ hare are (aj that one can discus, L|v 

1 riu? Ikw^ rCl#renC ^ ■" ^ « n ™«*W ™'« ene c .-,,.,1,,, reaijil5 }n fl „ 
.nE^ 4 t.«ni and f W th.1 51976 m^sl b* - n d e , e d in the S.F^ JLB76 undac the cnlry SHAPE. 



not TGRS.O-SUAPE (although it may as indeed u^der TQRSO-SHAPE »e well). This Is so that 
Olhcf Satoms can refer To it through Ihe reference SSKAPEi they do not have to know that it 
■ is a particular TORSO-SMAPL One important * } y m which template* c*rt grow mora 
specialised is by replacing general feferenee? like SHA a E with particular ones {tito T0R5O- 
SHAPE>, but 1his is a much laler issue. The act of inslanliation is a eomrnon One., and tens of 
Satomj are crealed fa describe even a. simple ^fjj, "The abundance ot Salome is one reason 
far tha importance of the T*f.erBncp-wirvdQW problem, 

Bu1 suppose one ask* 1st aOrne property of S1S76 that is nil specified en the 
TORSO template, but which is speclNed so^E-were. Far example, we might ash what 15 the 
thickness of Ihe 1ors0? This information will no! ba found unoer TOflSO Or SS87G because it 
concerns the SHAPE of the TDRSO, and so is one *t»p removed" from information that is 
djreetJy accessible through MS7$, This is tha difficult part 0* the property -inherit ant* 
problem, and Jt w#t not addressed by Rtphael <1$6S). Rather than trying to design a 
unive-sal solution to 1hi-s problem ike Fahlman (1975^ we look the view tnat the Only thing 
neeced to solve j| is knowledge about where tne reouired information may be found. This 
knowledge is held in the Sndev, a; a ipfccMicaliOn of which packets can Organize (in I hie 
example) THICKNESS, The Sinde* returns the reference SSHAPE, which mearts lhat m Order to 
discover the thickness of 31B76 we have to interrelate its SHAPE Salom. TJ this cHi&ts, il will 
already ba indexed under SlS76- h s SHA P E eMry. [f it does not already estisU One is 
instantiated. This causes a secdna call to tie Sindeif to discover whether a special shape 
template- enisle 3or Ihe shape of a tOrsO. !1 does, anc the Sindcx returns TORSO-SHAPE, which 
is the specialists internal name. This i-s instantiated, and Ihe required iniormatign is then 
found. The impprtar-l points here are \it the use of Ihe Sinde* 10 guide 1he search round the 
database (like PLAra\jEP. theorems fc and {h> once again, that using information irom the frndex 
involves a Srclerence evaluation which, if unsuccessful, can cause an instantiation. 

The nc/cr-r?icn-icJnJa« )\robli>m 

The relerence-winddw problem is important hetiuse (he mechanisms invoked. 
here ire what mjir.* it possible to apply one legate (Or scenario) 1o many different 
in±1 ar*es, For example, t HOSSE template will contain reterences to STQRSQ, which in any 
instance of that template need to evaluate ID Ihe particular SatOm for that horse's template. 
The scenario "VlflGlM SACRIFICED TO FEAR5CMt-TH[NG* will hav* inlernal references like 
SVJRGtl^ SFEARSO(W.E-THtNri t in terms of which, information in the scenario - the motive of 
appeasement, and the secluded ingestion 01 one party by the Other - is expressed. In a 
particular instance, the general statements cont^ina-d in 1he scenario must be transformed into 
specific PSSBTt-ions about Uary-J 3 ne and GodjiJIaf SV3s?G.lN must evaluate to Wary-Jane f and 
*FEAHS0WE-TH1NG must equate to Godzilla. 

The reference-window problem is almost the inverse ot the property- 
inheritance problem, and to implement it re^ui^es extra indexing. To solve it we {a) 
decentralized :he necessary indexing; by distributing it among existing Saloms (this Is what 
makes Shem into packets); (b) added entries 10 the Sinda* that help one lino Ihe local index- 
appropriate tor a given teitrancBi and (c) added extra lop-level access-points where they 



prated useful. W, have already seen several Sample* of ta) ■* <b>, and an example of f E ) |. 
thai when a H0R3E itnpto , 5 instanUatad, it* Salpni it ^tach^d to the Sinde* by the 1op- 
level reference* TOSE ard SANSMAL The reason for !h is is Jha| „,«!, kno^dge ab^ a 
HORSE is best represented as knowledge. About AMLULs, lor rr^ons Q | economy, 

. r j'iiH> ,i,.rti-;..; f c.-r i>.-n;jrd 
As we menlioned earlier, (he 3-D representations mat w<? use 4fe loosely 
hitrairhrcal. The hierarchy {such « there it) is expressed by the local .ndB.ing sh- U c1ure 

..■:J.I c;vr- S aDCu: b« £Uift ^tcn :.,n tfj-n : ;, ,, e I ■ , nntanlialicn * C H 9 - .finl,,-,* h.ut fh™ 

nei, .proved Id inde*. This ein najJBM rn severa , ways . One . E to ask the Sinde* t* a list of 

^ III. ™ ; 0dd " ' 1 ™" "^ * JiS ' ° f 1emptStBS ■ lhe PARTS Ql * n ™'"«. insist 
pF ih HEAD, NECk, TORSO, FtaELEGS> HlNDLEGS and TAIL The process of ACTIVATION » 

defined m Ihe inttHtratfen el th. PARTS 0- a Satom, 11 occur* whenever something import,* 
Happen, la that s atoni - for Sample if ihe Sato* is used Is sel w 3 sp^e-fran* In the 
image-space pra t «sOr - and il may he Ih^ht of 3E the lazy man's w Sy of iftstanliatinij * 3- 
[J model. 

The instantiation sequence is nOl reacted 1o using ihe PARTS list, in a special 
errcumstant., the pressor CM avoid the PARTS reerpe altogether, mtl«iti*Lm B only wh at is 
required by current needs for sample by spalially d^en infle* accesses to what is al the 
rronl of the torso or 11, rear, afc, Thus it can happen that differ! parts oi an animal sre 
currently represented a1 suite dHJerenl l evo i* o( d*1-,l. [f the front left hoof Is being 

Sl^l'eJ** Cher * |S "° '*"*" WhV th * * in<ileai *^ uld havfi l? " n unp^ked beyond a coarse 
HINDIS Dc SC rip:er. The hierarchy in 3-D model Conization is not strrtt, and the Ideal 
ordering i; not even total. For example, one can move straight frdm thi torso to ihe left 
,fOreie SF It is no! necessary to pass Ihrough She pair- f-forelees tlitailrueture The 
pardcutar ,oute that One chooses and Ihe amounl oF det.il For whi th cne instantiates 
aescrrptors depends upon the purpose for which the dnimal is hems, viewed, 

Th* revere of Motion also occurs; thai i,, instan1ialin E a jiv Bn Satom can ' 
«u» the IntUiitirtion Of a superior one thai tften indexes the Originai, This"!, so important 

Lul™ y * ^^ Un ' eS£ * iUhable ,ww,0r air " d ^ EKisl5 ' 0r unle » tJlB Sato ^ i* iheW * 
PHVSOB (wh.eh ha, a top^lev,! staM). Indexing is imports be,* U5S if an ilem is not 

Pl y( -Bd rnto the Sinde. somewhere, all r(Here«e* to it will Fail, and t,h c H Bffl will e-ssenlialiy 

have been lott tarev.r. for example, if a TAIL lempPate is inslantiat^, it will cause the 

,*stantra„on of *n AN2MAL Satom which 1her »t !Khfl s i1«*Ji to the 5,nde,, and i h ™,gh which 

relerences J,ke STAJL Or (5TA1L SAN[MAL? may b* evalualed EUEcewiully In thi, way * 

particular ta.l becomes connetied to She £ «ne,a] knowledge aoout animal, and animal tails that 

ra held .n Ihe Smd«. [t II intere^ing how a ^on E p^,„| p?oP er1y like cohe^vene« comes 

to t-e reflected in thi dataslructure as an indevin^ strategy. 

Before we discuss Ihe third symbol-mapping problem, that of indexing (or 
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FIGURE 12. Tht Sami and Sspasar are utftd to inlsrjret jn adjuncl region wilhin a space 
frame irt Ihe Figures above, the &axis is mamlained fin 1ha inCnhoy's tsriQ whiia the Sspaiar 
■& moved flntC' Ihe variouB *djundf */ Ihe torso. In {»), the Sspnr.ar is Bound to Ihe aisis aF 
the fr-onkey's head. 3o {b> and te) p it is bound to axas Jor eacti arm;, in (c> arid (d), 10 are* Jpr 
the Icgt, *rt<j irt (t) p T* Ih-t t»il. W* ?** he* etch p^r would appear from a particular 
orientation. 
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FIGURE 13. The monkey shown in «v*ral spate frame ori*nUt>0r 5 . Each image is praduced 
by •rtjpp^s qli: H |i D f tie monkey's insist, alec ; ,v C s while keeping the spa™ frame fi„ c d. 
For example lhe monkey's nos* i s i DU nd by slvtinE *ith tte Sjk.j on I he monkey axis and 
losing the lorw am W |th [he Sspitar, then tht Saxis can b« „ t to the Sspasar's position, 
fncini it on the torso, and the neck cm be [oca ted by lhe fspas^ us i fi j the I&rso-ne-eh 
adjunct region. This leap frag process continues until tr* S B pasar |] on lhe monkey** nose 
When the space tram* h related, Wi must r*t0n»lrijc( the entire imdg*. An important 
»r«| B ry to this is thai we only construct 1hose detdLI* we need »n<f 1hose only wtmn Chey 
art needed, [n (a} P {!>) ,nd fe} * slandfn, monk*/ is retard about Svirtical in 45 degree 
steps. 3n WJ P { t ) and {.) lhe monkey ,, s firsl tilled backward AS degree* jnd then rotated H 
abov*. 



recognition,. *t need an under standing at how Ihe pieces OF machinery deser-bed *o <ar work 
together. Let us therefore assume that a 3-D model has already been selected, and observe 
ho* it inlerFaces with ihB imase-tpacfr srocesso'. 

Suppose that 1he wtnmindfi {IN$TAWT]ATE MONKEY) and (EWTEFHWAGE-SPACt 
SMONKEV) are executed, The tfifrtl is to instance a. Safcom from the monkey template, to 
activate Jl (which instantiates the major parls of a monkey}, and to set up a space-frame for 
She monkey. The San is *$. bound 1o the monkey -atom, and 3 he ^spas** it nol ye! assigned. 

By executing (CHANGlE-SAXIS STQR£0>, and then interpreting the adjunct 
relaitions between lha torso arid the ani-mal's limhs, the Sspasar may he placed i.n any ot 1ha 
potions 5 nown in figure 32. By rotating the spare-lr^ne ahOul the four allo^od a*es, lha 
stick-monkey may he rotate* to any orienlatiOn (figure ]3). 

I! is important lo rcT^n-.DC" :-,a\ n practisa, only two vectors Ce.g. the Slorsc 
_ and the Stail} are represented at sny one moment. When the Sspasar is rebound to jinolhcr 
Jimb, record of the p*edided appearance of t«e previous one disappears, and only the adjunct 
relation that was read off the Ssoasar remains in Ihe 3-D model, The 1JJAGE.L properly dF the 
tail 1 * SatOrn will Mill point to 1he image element in question, and if 1his moves, (lags- will have 
to he set 1o warn thai the lOrmer adjunct relation may now be mvalid. We c*n reasonably 
suppose that in real life, IMAGEL bindings in1b a 3-D model tan be mamlained even while the 
Sspasar is bound elsowhere, because relatively low-level tracking, algar-thms suSfice to follow 
a moving ham in an image (Chien & Jones 1975, Soeckert 1975). 

Iltdtii)\ g far recognition 
Wo have seen how the 1heo r y n s mechanisms rurv in an isolaled slale, and we 
turn now to the relation between those mechanisms and the image during recognition, The 
interesting point raised hy pipe -cleaner animals (figur* I) in -the content of the present 
tnEOry is that one has Id use a 3-Z dosc-iclicn Iron ihe database before She image~5pac-e 
processor e*rv be rum but one might tNnk thai once a 3-0 tfa-scnption has been selected, 
recognition hw in some sense already taken place. In a full image. One might argue that 
"other clues" suiFrC* to seted the appropriate 3-D model, but in Ihe pape-c leaner model 1here 
are no other dues. The traditional A.L answer tD this dilemna is to hypothesize and lest, 
usi-.g error .nlo r mat-pn i-. some kind ol "dilfa'ence-cireded memory" 1b move to a better 
hypothesis (Mins-ty (Fahlman) 1575), This strMegy violates the principle of least comrnitmenl 
on which much Of our vision sys1em is bull!. Fortunately, [t is possible even at tins Isle stags 
10 design the tfjagnostrc system so thai il obeys 1he principle of Jaast commitment. This may 
be accomplished by uain 3 9 V n eraf animal 3-D model (possibly several - large, medium and 
small-animal), whose axes, are Only roughly correct, and which eilher has no functional 
semantics pointer ilself, Or Only a weak one. Using a general model, g.oacl e&trmales can 
UEually be made OF the actual lengths f inclinations &f\ti girdle-angEes present. By attoSEing (he 
index using this new information, (he correct 3-D model and it* functional semantics pointer 
can then be recovered, When Ihe corded 3^5 model is evenlislly lounrj, tb* OnJy visible 
change to the top-leva! organiiina SatOm Occurs in the value of it? SCLASS property. In some 
sense Ihis cnange ccnst.r^lgs Ire act d* , j et02nilior., because aF1er it h* s occurred, references 



v/iU automatic \y evaluale into the "recofcnriB-d" template. 

Figure 2 thu*ed * typ cal datailrgtture- that might have been delivered by the 
lower parts 01 tne vision system, 11 consists ol a set of axes., computed by segment irg a torni 
(Marr ft Vatan, to appear), bogi>d to each oF which is a property li l! (possibly' null). This 
property-list m 3 y ior example describe the shape of ;h P generalized cylinder whe !e a^s Shis 
is, and possibly mnu? descriptors af its- 5urf3.ce lexlure. In > pipe-dean e r animal these 
properties are irrelevant. Th# first problem >s to ace?:? an appropriate 3-D modnl from the 
dj*tab«e. There are varjout pa'ame-lors by which |h»« models may be indexed. 
Connectivity is not destroyed by perspective triinafornatiorvs, npr- are numbers like the 
fracliorial distance down one axis a1 which anathei aicis connects to it. Spurious conned ruMies 
can of course he introduced if one axis crosses in ironl of snolher, and if the reason is no) 
recognised lower down; but oxJsli^g enr-nedions sannot be destroyed, only obscured. Hence 
in order to use 1he connectivhy mformalicn, when meniunng which database items bast malch 
* given configuration set, unexplained errors pf omission are Irealed myth more seriously 
than unexplained errors of commission, 

The second sort of informality is girdle-angles, inctinalions, and the relative 
lengths of fines. [1 is easier :d lake advar-tap.c of k-ess laler on, when the image-space 
processor has delivered at leasl partial results about Ihe three-dimensional orientation 
relative to the viewer; :ju1 r, is possible to dp somelhing with them early on r This WmM 
aOQut through we a*, gross Clues. For oxampJe "vtrlitals" in I ha image are qflen close to 
verticals in real JiFe, and if the apparent length oF a "neck" exceeds the apparent length of * 
leg, and if bolh are qiulSe larje, the image fs likely 1q be a grafts. ]n Other words, lower 
bounds can alien be inferred,, and are SPmclimes useFul. Another jmporlant type of clue 
concerns majGr ditte rentes in (he girdle-angles oi t*o axes Iha1 are connected, to a common 
ene. For enample, \ht necK and the tail Oilen point in very different directions - one up and 
one down - and 1his obvious deference can be seen without a sophisticated 3-D analysis. ]n 
a pipe -clearer animal, tnis very rough differ dhcb can help to determina which end dI Ihe 
animal is which. 

Bearing these considerations in mind, we see that indexing cluea can be divided 
into two kinds; those that can be used before 1he image-space processor has befrn ^.A-nc into 
play, and finer does thai require it loast a preliminary guess at the 3-D conrigurar.ort bc-tore 
they become sufficiently re iable. Th c f^rrre- calegc-ry includes cdnnaclivily, fractional 
lenjjths on One axis, some corrparisors between the lengths of different axes h very rough 
relative gjrdle-ang'e comparisons belween 1wo axes that connect to a cpmrrt&n One h texture 
and roLgh shape irfcmat m, ^n<! pDssioly generai information like Che number of a*cs thai) 
are prohacily hofiion.tal> va.-lical, or naither. Such information in *ac1 provides a surprisingly 
rich body with wHch IP go to Ihe indexes and this is par] of the reason for our opinion thai 
Straightforward racOgnilion [recovery of Ihe lunctional iinrar.Uct pointer) is considerably 
over -determined in a nalural system. The- second CHlegory include? rvjch fine^ indfrKing on 
the relative lengths of differ-e-nt axea and their 3-D relation to One another. 

We can now follow the course OF the andyiis that lakes place when Ihe *yslem 
is presented with Ihe dalaslructure shown in fijure 2. Firs My, the connectivity of the axes is 
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. discovered, and ceded ready J ox accessing ihe indtxer. Neualia (1974) also described 
accessing an inde* using swh eennectivity information, The connective and the distance 
down each axis a1 which the nlher connects So it ^e bath use* Each join Is characterized 
using an overlapping hash-bucke! *y*lrjni that makes adequate allowance lor errors Of 
me-suiewnt, At the same time, She axis tor axes) with the mosl connexions are noled, and 
they form a possibilities list for the principal axis of Eh* structure. In this example, (here is 
np ambiguity, because She torso axii has many more connections 1han any ether. 

The call In the indexer reluma a general ANIMAL 3-D modal, togelher with a 
lis: of r. c .,-cral specie animal b. In real lire, 11 .$ Iifc a ly lhal auxiliary information would suffice 
to narrow this set of possioiiilies oown 10 a single candidate, hut here we take, the more 
difficult course Of using ih* general AM.MA. model. This 3-D model is instantiated and 
activated, and the SatOm for the torso is identified with the principal axis in the image. This 
identification- is nOl yet complete, because wa have yet 10 resolve its polarity {i*, which end 
of 1he animal corresponds la th head, and which to the tall), 

The pdarHy of Ihe torso is Pve'-determined in this example. ]l could be 
recovered from the Jail-cow.-. «-j neck-,,? ce fallen; or From the rough shape-descriptor* 
that ware happily included in the problem tlalemeni. Once the polarity of the torso is 
determined, tne system proceeds to re-express Ihe .m* e e elements in the form (origin, 
increment), ralher tn? n the form {endl, end?) 1hat was originally given, because lhal is (he 
Jorm needed for matching with the ima^e-spaca processor, and because if oOw has enough 
information Jo be able to do Shis correctly. The image element* are Jhen hound into tne 
1MAGEL properties of the Si torn* for 1he remaining parts of he ANIMAL- This binding 
includes backpointers, so (hat (for example) wre One leg to move and this fact noticed by 
low^eva] routines, such routines could interrupt tha higher structures directly ralher lhan by 
having to initiate a top-down search iflr 1he IMAGE! thaJ changed. 

The result of this process is the datastrnefyr-p B hOwn in f I jura 14. The system 
now instantiates a space-frame for Ihe animal, and is ready to commence the relaxation 
process that will result in the reprcsanlalion of Ht Ihrae-dimenj.onal disposition. 

In real life, initial access to a systole 3-D model may be more difficult lhan this, 
because the axes that emerge from segmenting a two-dimensional form can differ in an 
important W ay from the axes that are natural for the 3-D mndel. One circumstance in Which 
this can happen is when an imporUnt axis of an objecl points directly towards Ihe viewer/ 
For example, Ihe side view Q f fl buccal segments nalurally irvEo » generalized Cylinder 
descriplion. in *hich Ihe bucke! m represents a & a slice of a Wne, and th* axis is vertical. If 
ore looks at the bucket from above, one essftnlially sees two circles ,o:n*d by the sloping 
sides. The principal axis Of the buccal appears as a poinl frprrr this perspective The same 
Phenomenon is exhibited by the image Of a Jong, thin cone whose axis points nearly dire-city 
■way from the viewer. Ai 5C . wren a V i C wea object Is very close, peculiar distortions cart 
occur in the relative sizes in ihe image of its parts. 

In order lo access 1h c co-rect 3-D -ro-del despite fhe-se Obfuscations, some idea 



of depth his 10 be introduced Into lha analviis fee/„™ addressing }he 3-D model index can be 
«jeee»lul. 1* the case »f the bucfcal aaample, rtme p. f0c , 5S haa to r „ |izfl lM (he 1w(J 
circles might he separated in deplh, *nd that if they are, Ihey could he separate* by a 
considerable dislancs. The clues thai signal this are ofiin nuance* of shad™ and highli e ht, 
and Itus lead? ik 1o expect that much OF lhe snalysi^ ol h e h,tin E and shadow can influence lhe 
proce^mg a | eiffltMy this riage nf recognition. We tbdnk of She computers that take pl*« 
here at deployins lhe Sspasir fo cOnstf*! from She irrage apfimary 3-D mots* lhat consists 
at first of an axis in deplh whose circumscribin & surlue Is bounded by the two visible circles, 
jne! to which extra details - like hollowness, 1h» claiur* of one end of Ihw surface by en 
orltoBQMl plane, and possibiy I hip addilion d1 a cross-strut to account lor lhe handle - ere 
added Al some point during lhe construction of !his description, th* intexer is swtesslul at 
f.ndmg a nutch with a bucket 3-0 rnedaE in the Salabase. We do nol al present understand 
this an^ mD f# [W.dsaly, but wb have [•* feeling 1h»t one might have to abandon the principal 
0! least cOmmilmant here in favour of some kind ol hypOthesiee-and-test stralagy. IF an 
"unuS.ua! view" becomes a common view, it would becoma profitable (0 index th* appropriate 
3-D model urdar lhe special features thai Obtain for thai view, 

Inle.'estinfcly, WflrringfoA & Taylor (1S73J lound thai patients with lesions Jo 
tha eight parietal lobe were greatly mpaired wren eon^Onled with unusual views ot Objects, 
Such patients, who can recognize the picture of a bwk»1 in side view, are unable to recognise 
lhe Same buck*! when viewed from above; and even dthy that the latter could be a bucket 
when informed that i1 is. Th* aulhors cD.meien.lBd that u*to national lighting was as effective 
as. an unusual perspective in disturbing the performance ni SL ch oalien::., an'c they Luggcsted 
thai this friMt because the more stra. e h1 forward 2-D features are absent in these situation. 
■ These findings ware recently confirm fey Ca.-ey (personal communication) who agreed thai 
"unusual vhpwb" us ua J|y correspond to view* where an important axi; is lores hortenect. 

animation 
With the image elements properly bound to a 3-0 model, we can begin lhe 

rtiaxalian process. This is an incremental activity of adjuring lhe Saxis' and Stpasar'a 
onenlalions, S g,dad by ee^lrainta conjiibuted by the image, the 3-D model, and externa* 
influences such as e ravi1y. The Objecl is to find the corral 3-D orientation which will aFlcw 
Us to calculate the Irua lengths of the axes and 1heir relalive dispositions in saasc. This 
mformalion tan then be used to access a more spac fie 3-D ,n1erpreta1ion of tne ima 4 e. 

The constraints we have to work with are varied, and we have been finding 
new ones quite regularly. Meet of th*m cOnlribule information that restricts th B disposition ol 
One a*!* relative to another «™ti!r« in rathar comply ways. The major problem In lhe 
.relation (as*, mty 10 be in umbinins fh«e incompleti clues to deduce lhe 3-D models 
actual orientation. Tne image-spate processor is turrenlly th. major f M u£ in lhe rela^alicn 
theory that is developing around this problem. We 1n & l (hat th« prvc&nar C3n be used 
effectively as a dynamic model of spa™, where the j M | E and Sspasar are used much as a ^hitd 
plays w.(h blocks to see hp* thay can S o together a iv*n the addilionaf consb-ainls ha wants 
to impose on the coniJ E uratJon r In our case, the tmis and Jtpasar are held togelher *S rigidly 



W it necessary to maintain whaler adjuncl infer-malion we hive available, and lha whole 
configuration ie pus had around unlil ils projection pnlo 1hB ima&t p ,-ir* ncil - o^ly ™k^i 
the image *c arc- iryipj to interpret. Kn C wlec fl e of these constraints and ?J methods for 
apply in$ them wiN be represented procedurally in the relaxation processor currently being, 
developed. In the discussion th*t rollows, we define some Of the orientation contents 
available to 1he relaxation processor and ho* we plan to apply (hem. 

The first constraints on an axis' DrienUNOn are calculated dlreelly from the 
image. The Intfal B«m us the orientation end len E 1h of Ms projaciion onto the image plane, 
SO She Only additional information nisded, to «rr.pute the axis* orientation, is ihe inclination 
this *xlt makes with the viewer's Ime of Bijjhl. Sometimes Ihis anr>ie can be estimalad directly 
from the imager* B hape descrrplion For example if the axis is known to be cylindrical and the 
intermediate visual processor ha* supplied a shape description sufficient 10 calculate IN 
lOcalion Of its perspective vanishing poirit, the axis will be parallel with a Ime From the viewer 
to this vanishing point. More often, the ima;ef h s shape property and our confidence in what it 
snouLd be wil[ not allow more than a very approximate guess at lha vanishing points location, 
so o1her cities are ne-s-dad lo coj-,slrain the axis further. 

The influence 1hat gravity and Olher external (actors have On the distribution of 
Image I Orientations proyide* addilional information abou; nttividLal image It. For example if an 
ima,gel is close to vertical in the image plane, there is a high probability that lha 
corresponding *,rj; „ vertical in space as Wilf, This clue does no! require knowing anything 
abou: ihe nalure of Ihe observed axis to be used, but it attracts most confidence w hen the 
axis is Known 1p be vertical in its normal & tate (e*. a horse's teg). Another clue has to do 
with the ground, [( h often flat, and when it supports the objects we view, it serves as a 
very strohf; constraint on lha likely orientations of some of their ai.es. Consider for nk.nnr.ie 
a tow s i torsoj it is very difficult tar the cow to hold it out of the (round plane without 
bending lus ,e S s Out of paraliaL 

When individual imaged fall to provide enough information, wb can look at 
several together. ]n the example with the cow, it was important that its legs were parallel ior 
insurins That h«s torso was parallel to the ground. Parallelism is a very mce property in that 
:he .rrarjels Kat cDr-espond 10 paralel a»ts are also parallel. In animals, Ihe legs are cjuile 
often close to parallel and this is a powerful means of disambiguates them from ether 
adjunct* Of the torso. Accidental alignments tan produce parcel imagels where the axes are 
not par*ll*j, but rarely will this happen for more lhan two axis at a time, and even this 
possibility is infrequent enough to [utflfy paying attention |rj * h y parallel axes that ere 
press n1 in the image. 

When or* axis has oeen fixed in spar.*, tna dispositions of its adjunct axes are 
heavily constrained and id Olten determined uniquely. ' It is a simple matter to rotate the 
Bspas*r about the known axis until its projection matches the image I Ol the unknown axis. 
This is what was done above wJ*n 1he animal's torse was rotated about the vertical until Its 
projection matched 1he torso amagel. 

Finally if no axis is delermiived but two adjunel relations ate known between 
three nonparalJet axes, then (except lor a few patboldginal cases} the disposrtiOns Of these 



axes, ara determined. The image-space processor can be used to discover 1he&B dispositions, 
thrash a process of experimenting wi?h various Oriental ions of tt*w three axes (two *\ a 
time) attempting to maintain lha Adjunct filalioni wh.l* minimizing Ihe discrepency between 
tha imageis and the projected axes. 

Let us now -*Uirn :c lha animal mage we hav^ bean processing. [J was left at 
ttie pair! whefe its 3-D model had been activated and enlered 1*10 She image -=p<i<;e wilh the 
imagel* CPfreelly bound 14 its axes, 51 ha* no verJitaJ awes, but it is Known to be en The 
ground and lis torso It very tlKely to be hpn zonal. The imagel shape informal ion is no) good 
enough to calculato any vanishing points, and only tr adjurcls bplweei :he !n<z<, ?ind 1ha 
Jegs are reasonably certain since (he others wary too much from one animal 10 another. First 
the S#xJ« is sat EO Svertic*l and Ihe Sspasar .s placed on (he animal 1 * torap adjunct. Thus (he 
Sspasar .$ forced to be perpendicular to lh» gravitational virtfclL The S S pasai h s projection 
musl ee aligned wilh the torso i.-nagel so lhat Ihe 1ree enc ol Ihe Sspasar is on. the tail side q1 
Ihe imagel. Onca Shis, ij dona, Ihe Bspaaar is oriented as Ihe animal's torsp is, PVexl the animal 
3-D model must be rotated about its torso axis so thai i1 corresponds with the image. The 
torso-leg adjunct is the most reliable, so the 5a* it is moved to She Sspasa^s current poshW 
or tne torso and the Sspasar ia put onto onB Di the Jegs. Again 1he Sspasar is rolalerf about 
the S*!fi* until its projection onto the. image piane r alerts the corresponding leg image!, At 
ihis .ioin: Ihe animus or.enlation is detB-mn ned. The naxt Lask is to measure the remairin£ 
adjgncl relations, A vary important constraint fqr domg Ihis come J- (rem the fact that an 
animaTs axes tend 10 be parallel to Oj*b plane. This m Bsn s that the girdle anglss oF the 
adjuncts are the seme module ISO degrees. So the inclination of the animal 1 * neck can ba 
Found by putlinj (hB Sspasa* on the torso-neck adjust and varying the inclination angle until 
the Sspatar's projection Unas up with ihr neck imageJ. Finally Ihe relative lengths oF Ihe axes 
stb measured by selling the Saare and Sapaa»r onlo two axes at a lime and adjusting 1heir 
lengthy until they jusl cover 1he corresponding imaged Figra 15 sketches Ihe relaxation 
sequence described above. 

Ninipiizinf tltr. r.amfrloxHy */ the im&gt*-t.paiL<! jirraeriipr 

In our account Of this Iheofy, (re image-spate processor has bean pared down 
to almost its mnimal implertien! alien; oniv si* directions art represented, only > w « are aclive 
and (a= * corollary) erJy Jour relations allowed. The original motive.) ion for (his was to see 
how simple a processor ceuld support |h 5 meEhanJsms that were required. Provided I hat 
coordinales are chosen as defended in figure S. (he computations needed to support the 
image-space proc ester are straight forward, and if is unlikely thai finding an economical neural 
representation for this pari of the theory will prove vary difficult. 

The main characteristic of a minimal implementation is lhat i1 should contain 
only one Votalable" etem C n1 h and rather Few passively stored directions. ] n such an 
implementation, a construe! in the space-lrame would nol survive rotation pF that frame, and 
would have to be reconstructed Thrs is quite a slrOng signature pf 9 minimal implerMnlation. 
The ability Jo hold direct ions in pas^ve storage would sometimes he useSuL (hough some Care 
is necessary when codding whether 1hey are sli.l reliable. 
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FIGURE IE. Once a 30-mOdel is selected far the oojetfj it must be related lo She dispcsiJicn of 
th* image so that the object's a*is lengths and adjunct relali-Ons can be measured. This 
pfEKeii \t -Carried nut an the example shown in Hgure 14. (A) jhowt iho 3D-rnCjdef in ils 
Standard orientation. Tp- gel the oriental ion of the lorso, an as sumption is made that it .5 
parcel to Ihe grO-jnd. The Sauis is set to Sverlital and the Sspasar in (B) ij set perpendicular 
to it jn th( imaga plane. The Sse-«ar is Ihen rOlaled abnul I he Sa^s unlil its p-r&jittj&ri aligns 
with 1he tar$o rmagel ir (C) es^soKsrilrie the diretlion fll the 1orso axis, Went the 3D-moGei's 
rofaliein about this axis mutt be discovered, so Ihe Sayis is movsd !p Mc fl^pssar's pas'.lion on 
The torso and the Sspasar is placed aa Ihe torso-Ice, ?djunc1 and rotated about the Sani* until 
it* projection aliens, with (ha- ipg imagel (D> and (D, At this peinl ihe 30--model is corredly 
Srienrsd. In- (F) and (G) the inclin.i1 ion angie of Ihe Sorso-neck fidjiuTOt relation is, mfiasuTO-tf by 
placing Ihe Sspasar on the neck axil and adjusting the indiraa,tion unfit its projection aligns 
wilh the necx imagel. En {H) and {]) the lengths 0' Ihe 1orse and necJ; are compared by 
lengthing the Sipasar until Me projection if !h* $amt *i the neck image I and shortening the 
Saxk tp irtateh 1he 1nrsa im*£elr 



Otherwise, the slruetura oflhe theory | u rn* less on th* constraint ol minimal 
complexity thafl One might a1 iirtt sight e *pect. For esarnpke, 0riB import character,*^ of 
our representation it its ability Id move lluantly fr 0m a c&arse c, n e-ax is description of i 
whale animal la a very line description pF one snal! part. By removing th> minimality 
constraint, one could easily design a machine ca D abie or maintaining a fine description of ill 
parts oF the animal simultaneously, but whll would be th- point? The Fact is Hut tor many 
purposes, a complete 3-D reconstruction oF 4 physical object is no better than the Original 
object. Ho* do you answer j n which direction 1 horse is poking U „J C „ p flrt of your 
representation contains something like a lorso ajfc? Many of Ihe diagnostics and constants 
!i--t ;ip:iy «;r. l; riL^r ' :;,-, ,, >: rn *isi=c- :.0:-,-- |,- d cvn'jll di&p&u.tiOn nf Sle whqi, 
animal., no! very local del ails (1 hough they tM cba he import anti The same is true of the 
questions lhal one a*k s of an image in real life, ] n a very luxurious impleimmtatmn, one might 
maintain complete description* at all livela C F delail simjltaneousiy, but the r^pyrces needed 
to do so co-utd probably be better employed in other ways. Provided thai a simple 
implementation tan Mmpute anthers to the important questions reasonably quickly, there it 
no reason to use a complex implement aliens 

HixGiLtflttn 
The discussion tails naturally intD :*& pa- Is. ore concerning tha specific 3-0 
representation theory, and the other dealing with 1he broader issues raised by the 
interpreter icn- jhal theory. 

There are fi*e main points (0 Our theory. Thoy are: 
(1) the 3-D disposition oF an object is represented primarily by a stick-figure conjuration, 
where etch stick stands for one Or more a*e$ in the object's generalized cylinder 
represent ation, 

£2J This COnfie.urs.tion is descried by a loosely hierarchical assertions! database, called * 3-D 
modeL Use of this database is. extremely free and flexible, and it can support level & of 
desertion that cover the spedrum frorr very coarse to ve^y fine detail. {MB. the principle 
Of graceful degradation.} 

(3} in offler to be useful, this database his to be interpreted through an {essentially) 

analogue mechanism, called the image-space processor. In its minimal implementation. I his 
processor maintain* a replantation of two diretliom; in a spict-irame, in addiliOn to the 
grav.tational verticaL, 

(4) The image-space processor's instruction se1 is tm*IJ. ]L& most important features are: 
(a) the ability to interpret an adjunct rei>li«n ae:wee- lh c 5av5s ard the & ED asar F and 
(W th* ability to execute lour frame rotations (about Sup, Sfron!, ^horizontal and 
Sverticaf). 

(5} The image-spate processor c*r< deliver intormatrdn about the lengths *nd Orientations of 
the appearance Of the fcxis a-d Ssp a5 ar. These n*lp (he system lo rotate its model into the 
correct 3-D disposition relative to the viewer, 



It haa not escaped our not** (hat this theory may illuminate certain recent 
findings in erpenmerita! psychology. Shtpa.-d « Vizier U971) created a set oJ ma%BS by 
rollting and reE^ctm^ simple objects made 0< cubes f.Jig Ura 1&J, They found that the lime 
taken to detrte whether two sucft images *ere of identical objects, ralher Ihan objects that 
diftei-ed oy a reflection va T i fl d linearly with the angle through which one object must be 
rotaled in 3-space tp become alig^d with Ihe piher. Thk finding revived ifllerest in "mental 
imagery" and in analogue processes in pefcaolion (Cooper fir Shepard (1973} Meljler £ 
Shepard (1974J, Shepard (1375)). [n addilion, Kosslyn (1975) ha* published evidence for in 
analD & ue component to the processes that inltrpre) miirdy Iwo^dirnensicnal structures, tike 
tunes and map*. 

The significants of such experiments I* controversial {but not I he results). It \m 
a commonplace thai in Observed description Of what h, does not understand is often 
r».*l«din B , betJuse the concepts though which h* attempts to capture She experience are 
inadequate. Because of this, "me-nldist" HK perirr* n !s and espetralfy the introspective reports 
that accompany them, are rightly regarded wi(h suspicion. Although it is widely recognized 
that a complete thepry ot mental processes wilt evenlualty have to explain Ihe 'incfin B s and 
th* subjective experiences that accompany them, we are probably not alone in feeling I hut 
One should not rely 4n their help i« construct a tneory, because of tne possibilily that such 
reports wilt become accurate arty after Ihe observer understands the processes that are 
givna rise to them, 

Another reason for lha -controversy seems to have been the difficulty in seeing 
how *n "eneioaue- process could benefit the corr-pulalions that underlie perception and 
recOgml.on. We be \^ that :he present Iheory shows * way in which such a mechanism 
could be useful, aitho^h we retog.-.ire lhat 1hi S - way not be Me way in whrth we En .act use 
il- Tn Order to help detic* Ihis, one orobabiy has 10 study (he neural implementation of the 
roechamsms that we described, in the hope Of making les table prediction* about a-ingle twit 
responses. Since this is a major undertaking. Dn e needs aome evidence that Ihe Iheory i 
indeed a likely candid*]*. There arc- several points 1hat appear to u* to «ns1itute re^OnabL 
grounds tor belong ihe theory to he a good candidate for a psychological thaOry. They are: 
fl) Pipe^leanar animals are almOsI as easily reco E mrable as are line-drawing OF aoim,l £ 
despite their very ibstfacl region to the Original. Tpus wfluld not be apprising if pipe- 
Cleaner ammals were in some sense extracted from !he irr,age during Ihe norm* course of its 
interpretation (as our theory K „ rU ), but il would be Surprise if r*l. The computational 
advantages ot so comg need re e^phas s. 

(2) The loosely hierarchical structure of our 3-D «od & |s has many computational advantages 
that are almost bound 1o be shared by hhe psychological representation, B ven if the 
penological represent alions nt* otherwise v.ry diFierent. One cm probably rule Ou| any 
system 1hat cannot restr.ct Ihe iev*I ot descriplive detail al an y point to only that currently 
neeoecL 

OJ An important part of the theory it the minimal nature of Ihe im*g -space processor A 
consequence pf this i» that after e*«tuting a rotation, the 'image" of the 3-3 rr.odel has to be 
reconstructed in the new space-Franc, as opposed lo be-ng constructed Once and then heir* 
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related as a whole in tha ima £ e SJ-ace. The I si Owns "menial experiment" will convey Ik* 
intuitions behind this to 1hB reader, WB have «nfined ihem Id the discission, because w * do 
not regard sych evidence as admissible In the debate about the psychological correctness of 
the theory. 

(a) [m* g jne £ horse. [For mist people, it is fating Hither l«l| or righl, which wb 
inlerpret as (he inilial space-frame canfiguratiflrx] 
{b) Imagine rMsting "he horse 90 decrees aboul ils 1orso, 
(c) Where is the necK pointing? [ttas] people Cirt answer this e^ily,] 
<d) Mow -magine a new horse m tha starting configuration^ only Ihis particular horse 
has his leg* glued 0n ,t the top of his back, pointing upwards. His he*d *nd neck are . n 
the usual potion. No-* rOtale the horse :SQ *e B ree E about h«E torso. Where are his 
legs pointing? Wher* t6 tr » neck pOmling? [We lind that people eilher "leave the 
neck behind' in this rotation, Bnd have io reconstruct ,1 af| C rwards 3 at jhey find 
themselves in;errupLing the reeOns-tructiDri of the legs titer the ratal ion because- this 
reconstruction differ from the normal one whi^h they had <by habit) initiated. We 
thin* of ;he aclual rctaSiOr as having taker, place with the 5a*is bound (o (he'torso, 
and the Sspasar either to an "unoercarnage' &&\ a% 0r | D a Salom for the fOrete E s or 
the hir.dlegs, or So a SalOm for Ihe r*tk, or head + nee* ("busl").] 
(4J The number of possible rotalians in our rrodel is sunl, only four being allowed. [This 
seem* to be true of mental rotations. Few ei*mp| 8j imagine a normal hors-e on« again, ™hj tB 
tail is about 20 decrees to the vertical. It seems So be straightforward to execute mental 
rotabons about Ihe 1hree principal a*es {[h* horse's up, frenl, and horizontal )t but not about 
arbitrary di red ions. For example, 1ry rotating it abOul its fail One either fails, or h« to 
re-sari to a special sh-al^em. H hoover one taegina-s ihe horse slanding on a 20 degree 
slops so Iha1 the lail actually f,U s down the gravilat onal vertical, It becomes easy to imagine 
rotate it about Ihe vertical - |,». about the Jims axis relative \q the horse that Was 
previously so difficult,] 

i5) The 3-D model is loosely hierarchical [In Ihe previous eMmpte, one might have thought 
that one cuuld move the Suit 10 the tail, (he Bspasar to Ihe lor», ar«J Ihen rolale the horse. 
This possibility would be eadvded it the tail SatOrn contained no adjunct relation lor the lorso 
Adjunct relations are npl symmetric, and Ihis is whal in our theory produce* tha diredional 
property of the hierarchy.] 

<&) There is a general agreement between Our expeditions and the results surveyed by 
Shepard (1975). Only 0raf at the Endings (Item 14 pa^e J 00) is unexpected. [\ comes from 
Cooper * Shepard (lS73o fondftiOn D), who showed that advance information e iv.ng Ihe 
Orientation bul not t.-, identity of the object 10 be pressed is not sufficient \o enable 
objects to prepare for if. One might have Hptclrt 1hat subjeds could rotate their fepasar 
to th* appropriate orienUtion, anp leive it 1her* to be bound to a 3-D mode-l when the image 
was prosenled- [n Order to Jncorporala this finding, wt woulci need to assunw {lor example) 
lhat Iho Sspasar machinery cannot be run unless bound to a 3-D model (even If only Of an 
arrowy and- that whenever the Sspasaf is rebound to a new 3-D mode*, the image-space 
processor is reset. There are some other grounds for wanting this. The spate-frame in the 



rmagB-spacB processor needs more rh a r one dirttlion to define it, and trying to construct a 
sp^ce-frame «und a. given VBC i ar cjn ,, BQ to p , 0o i Bms if the g_ D mo(J#| ^ m| sjmp , e 
Secondly, in the real w8 .-ld p one rarely sees We Objects a) Ihe same point in the field of view. 
Therefore, to charge 10 3 ne* 3-D -rwSdel almost always requires a change rn the direction of 
gaze. En order to componsale for this in a mini™) implementation^ the $axis and Sspasar 
should h*V. to be sat to axes in She sluing ffHme, in order to M riy Qui the primary rotations 
(hat allow (or the *n E J B of B a? c , These argents are howwr wea^r lhan the argents 
that Support the rasl of tftp Iheory. 

The reader »* amuse himseH Dy construing menial relation problems, and by 
devising strategies 1o answer questions cm which he fails the First lime. By noting and 
exploiting e*tra rela-LiOns between Ihe »*,[& of a sfruelura, one Can quickly become much 
mere versatile at answe rir.fi ques1i0ns abogt Ihe sppearan CB Of an opject when rotated in a 
new w flYl ][ appear to us thai tha mechanises contained in our theory cm accounl (or me&l 
Ot the experiences, that one has when imagining iuth things, and 1his is partly why we find 
the theory interesting. But we are perr ec tty aware that this kind ol evidence cannot Establish 
that the theory provides J correct or *Ven an adequate model ot Shis component of our 
perccplual faculties. What we ^e ctaim for tha theory is thai the c-ompulational facilities it 
describes, are useM for recognising, and representing the disposition Of an object in Ihree- 
dimensional space. 

Si Broader iitpv? 

It is no ttcidenl that the tajm "frame" {in the sense pf Uinsky 1975J hit not 
appeared in this article. We have been careful lo use Only technical tern* (Satom, template) 
that have a praise meaning in Our theqry and are supported by a wooing program. 
Nevertheless some of our ideas h*vs been inil y e«ed (positively or nagalively} 6y MinskyV 
extraordinarily stimulating < an d frustrating a-licle, ar,d wc must attempt to reia1e our work to 
the ideas oF frans theory. 

At the very top level, Mmsky musl surely be correct ^hpr i e Disserved 1ha1 
the "chunks" nf reasoning, language,, memory and perception ought to be larger and more 
tifufilurod than most theories in arliljcia.l ln1elii Ee n C a and psychology allowed. Failure So 
realise this, led to absurd attempts tq "prove" from predkate-c-alculus-like "a^ems" 1ha 
"correclneas" nf 5 tr 3 | e gies fa- BMS i-. s between Iwd rocr* 0' crcunnavigaling an ebslacle. 
Reaction to thai iine of thou £ ht led 10 the procedural embedding nf knowledge, to now 
linkages like PLANNER [Merits 196 3) , n d I hence to C0NN2VER (Swsman & WtDermOtt 1972) 
- a vaiuabla axperimenT that ft K run sufficiently long np w for the results to be in, *nd Ihey 
are contiu=i./c >■ -.- La t,vc- Itr.sus.- ■* t:- much pss^ivB spin-ofl). 

Escept at this very -jenaral level, i[ is not clear that the Ideas of Irame theory 
aft relevant to the spettfhc problems ol visual perccplion. JJin 6 *y himself has made no claims 
= hat Sra^oes are relevant To early vis wi f information processing. The grounds for extending 
this conclusion to Later processing are as follows; 

(I) CvMtilt phenomena. It is probacy incorrect to thin* D i f fimc theory as being importanl 
for perceptual Geslalt phenomena. The Kanizsa lriar s | fll ^nd ■jsun"' illusions like figure 9a 



of Wirr [1*75 a) are no! caused Py the descend^ inf luences of * high-level -frame-like* 
or B * rt . E *l.on Oi Ihe pertepl; rhey . arc CUB fD 6iri|tra |.p utp0w infermedia[e .| e ^| er6upins 
processes that act on th* primal sfceith end together perform much of fijur*. ground 
■•partition dee also W*rnngt D n & T ay | Df 1973 p. 15fl). Recent work by & UlftWI (pErsonat 
communis aim n) demons] rates thai the s^e is true for motion vision. Fi £ ure-ground 
separation by relalive mc^on is net caused by extensive tap-down matching from a "frame" Lo 
the image. It |$ due almost entirely ta local rnakh.ng processes thai nperale on th H chafing 
primal sketch, and it 1&K*s plite before any descriphpn of the separated figure it computed. 
Srmilar remarks hold Tor percept* due Only to. binocular disparity information {Jules* 1971}, 
and far the rocOgnition of symmetry In a figure (Marr 1976). 

f« J/nh^f* r««a rep^^if™. [t is d-HJcuM to *rg gB cogently a g B j ns i this represent, I ion, 
hecauBe it is at present unc:erdaFi-ed - i r example, *r e all M ews - oF a min thH 5ame ifl 
which the sane limbs ara visible but arranged in different pOsilions? Nevertheless, sOme1hin S 
Of a case aeainsl it can be made from Warrington & Taylor's 0373) findings. The. side vmw Df 
a bucket is very diffWil from the top view, and coin ar B reasonably simple. On B would 
expect the multiple view representation to contain them both, and {presumably) tc have 
indexed both of ihem. ]f Warringlon ft Taylors Jetions had randomly damped Ihe multiple 
view represent at ion, One would aspect some palie„|s to have lost one view, and Others, 
another. Bui Ihe finding is lhat all patients are impaired pn th* same view, and thai for this 
and Other objects teg. a clarinet), the los! views' are precisely Those Furthest removed from 
the Objects' generalized cylinder representalions. Although [tie multiple *«w represent alinn 
is not absolutely incompatFbie wilh these findings, strong- «x!ra assumptions are needed to 
-corporate them. On the Other hand, 1hey a» a natural ennsequsnee in o„r theory of losing 
"he image-space processor. 

(3) Ft*t** itttoohfy. The major difficulty in clicking other aspects of frame theory It thai 
they have not yet been made specific enough to be refutable. Fur simple, exiling 
cxpo&nions OF 1he theory fail to deFine what "frames", "terminals", "slols" and 'swidlc nets" 
could aclually mean beyond Ihe old ideas Of property-lists, VB [ ueE and dcFa(J ,ts (11*6 
Raphael's), and some sort gf labeled graph slrwrfura - all pf which are useful ideas but which 
are too simple to carry Ihe load atlachad to their rotes m frame theory. We encountered on* 
problem thai might be related 10 the nolion Of -terminals" on a frame, if the -par| E H of one of 
Our hor s . 3-D mode-It are somehow canned as trfnskian terminals On a horse "f.-aia**, then 
On* can perhspt make sOJne correspondence bciwaen entries in our packet indexes and the 
lerminals ol lr*me theory. The analogy is flawed, because our packets ,ire based Pn 
reference evaluation not on matching { fl distinciion we regard aa crucial), and in any case wb 
found lhat fixed terminals proved too in1lfl*ible to be useful. We needed a system in *,hkh 
the?* were many ways of describing the part? pf an animal, and Ihe particular -nnes chosen 
(e.3- fOrele GS , or lefl^preleg and righl-fOreFej, Or sometimes bo!h) depended on the 
i-rrcumslances. The real itsue, as Fahlman grasped early on, centers on creating this 
flejribilMy, wh.ch in turn places the re'erence-^irdow prppiem and its associated Indexing 
straits firmly at the center Of Ihe debale. Li*e Woods (1975), we fognd the idea oF a 
semanlic flat- too vague to be useful; differential fcignOsia btsed On a difference-direc^d 



f«*W V villas Ihe pr.nciple of lert «m mi |« Mlj ,„ d dpes not app«r to be r.fevanl |* the 
type nf re-eogrudon that w B have been sludy.njj. 

^W^cmeaUi 0. U. ihank* Dr. S. Banner F. ft, S. for i^orfanl converses m 1972 
.bout rtferftnc and n«t«n &J Ken Forbus who helped *r,|e MEMNON, an ** p0 rim e ntal program 
^rrtten over the r, 3 f IS months 1« develop 1h, pr ii«ipl« On wh.ch our i fl terp,Bler is b* SE d- 

>nd Drew hfeDe'molt tor his perspicuous in d psnetralin* crrlkis-n, We ^■:>--,„^ f WB 

COulrf nit have developed ih, theory wilhout tl* e*peTie*t* of i« P Wervtin S i(. This m | U rn 
would MT have been pPssih* y,iJhP U 1 the eatensivs ™d n-Xffafe tompuNrg fadlilies that a,* 
available *( 1 Hie laboratory K«ren Preoder^r prepared lha drying,. Work sported h rein 
was COnducied al I he Articiliil I^INgante Lab*ra(o,y, a M« s „hu«H, Intitule of 
iechnp.o*y research P ro E , a * 5U pppded in par! by the Advanced Research Projects Agency 
Of Ine Dewtmenl nf Defense and nrtnilored py the OMice of Ma va | Research under Conlracl 
number M30Gl*-75~C-0643. 
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